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Preface 


The  objective  of  this  6.2  project  was  to  transition  tests  developed  as  part  of  the  6.1  Learning  Abilities 
Measurement  Program  (LAMP).  The  work  was  conducted  under  Inhouse  Work  Unit  1 123-A1-19,  Transition  of 
LAMP  Test  Technology,  however,  the  project  was  cut  as  part  of  the  APOM  FY99  budget  reduction  in  the  manpower 
and  personnel  research  areas.  This  paper  documents  the  Advanced  Personnel  Testing  (APT)  program  which  was 
designed  to  enhance  Department  of  Defense  (DOD)  selection  by  assessing  new  tests.  Given  his  overall  familiarity 
with  the  program  (as  an  outsider)  and  his  technical  skills,  Mr  Scott  R.  Chaiken  was  asked  to  edit  documents  into  a 
serviceable  technical  report  for  this  project.  While  these  documents  were  never  intended  to  be  final  technical 
reports,  it  is  his  opinion  that  they  provide  the  best  available  documentation  of  what  the  APT  project  was  and  what  it 
found. 

The  first  document  in  this  technical  paper  (Section  1)  is  a  presentation  at  the  1997  American  Psychological 
Association  (APA)  conference  by  Goff,  Sawin,  and  Earles  that  compared  the  factor  structure  of  the  APT  battery  to 
that  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  tests.  The  other  document  (Section  2)  is  a 
briefing  given  to  the  Office  of  the  Assistant  Secretary  of  Defense,  Personnel  &  Readiness,  Military  Personnel  Policy, 
Accession  Policy  (OASD  (FMP)  (MPP)  AP).  The  briefing  is  a  necessary  addition  to  the  paper,  as  the  nltimatfi  goal 
of  the  APT  project  was  to  assess  the  incremental  validity  of  the  APT  battery  against  a  final  technical  school  grade 
with  ASVAB  already  in  the  prediction  equation.  That  question  is  only  addressed  in  the  briefing. 

The  APA  paper  is  provided  “as-is”  except  that  the  original  paper  references  three  figures  corresponding  to  an 
APT  model,  an  ASVAB  model,  and  an  APT  +  ASVAB  model.  These  figures  apparently  were  never  completed, 
however,  Mr  Chaiken  constructed  a  single  figure  representing  the  most  complete  APT  +  ASVAB  model  as  a 
substitute  for  the  three  figures  (i.e.,  all  models  referred  to  in  the  APA  paper  are  subsets  of  the  model  implied  by  this 
figure).  The  briefing,  for  which  no  slide  notes  were  available,  is  presented  in  excerpts.  Slides  pertaining  to  the 
origins  of  the  APT  battery  were  removed  because  this  is  covered  in  the  paper.  He  also  removed  some  of  the  backup 
slides  that  seemed  to  follow  a  different  agenda.  The  remaining  slides  address  the  main  results  of  the  APT  project-- 
these  deal  with  relative  and  incremental  validity,  as  well  as  test  fairness. 

Both  the  APA  paper  and  briefing  were  interim  reports  given  before  the  data  collection  was  complete. 
Therefore,  Mr  Chaiken  augmented  these  documents  with  results  originating  fi'om  a  more  complete  data  set,  having 
processed  the  data  following  the  procedures  of  Goff  et  al.,  except  in  two  cases.  First,  he  reports  results  on  data 
corrected  for  range-restriction,  given  the  prior  selection  of  study  participants  on  the  ASVAB  (although  both 
corrected  and  uncorrected  data  are  provided  in  Section  3  of  this  technical  paper).  Second,  with  regard  to  EQS 
models  investigated,  Mr  Chaiken  set  the  variances  of  all  factors  to  1.0  and  allowed  all  loadings  on  factors  to  be  free 
rather  than  (as  in  Goff  et  al.)  fixing  one  factor  loading  (i.e.,  the  highest)  and  allowing  the  factor  variance  to  be  a  free 
parameter.  The  validity  (e.g.,  significance  of  the  results)  he  reports  would  be  the  same  as  the  Goff  et  al. 
representation,  which  is  not  an  “identifiably”  different  model. 

Sample  1  is  the  complete  sample  of  Air  Force  basics  who  took  both  APT  and  ASVAB  tests  (n  =  2,270),  which 
contains  all  tests  plus  a  final  technical  school  grade  for  the  security  police  specialty.  Sample  2  is  most  relevant  to  the 
briefing.  The  range-corrected,  or  disattenuated,  covariance  data  for  these  samples  are  provided  in  the  context  of  an 
EQS  specification  for  the  most  inclusive  model  reported  in  Goff  et  al.  In  general,  parameter  estimates  and  fits 
reported  in  the  paper  are  very  similar  to  the  model  fit  to  the  full  sample  (allowing  for  the  expected  effects  of  range 
correction).  Sample  2’s  model  is  the  same  as  in  Goff  etal.  but  adds  one  variable  to  the  system  of  equations,  namely, 
the  final  ^ade  on  all  the  factors  present  in  the  ASVAB  and  APT.  In  addition,  Mr  Chaiken  used  SPSS  to  address 
standard  incremental  validity  questions  (by  analyzing  Sample  2’s  corrected  matrix).  The  results  given  in  the 
(overall)  abstract  are  derived  from  these  analyses  of  Sample  2. 

Some  results  discussed  in  the  briefing  but  not  addressed  in  Mr  Chaiken’s  archival  tables  (i.e.,  issues  of  test 
fairness)  could  be  addressable  in  the  anonymous  subject-level  database  that  generated  the  tables.  A  text-based 
version  of  this  subject-level  database  can  be  made  available  for  academic  purposes.  Unfortunately,  data  pertaining 
to  schools  other  than  security  police,  as  reported  in  the  briefing,  were  not  archived,  though  the  results  reported  in  the 
briefing  concerning  the  other  schools  appear  similar  to  what  was  found  with  the  security  police. 
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Section  1 


The  Factor  Structure  of  Two  Abilities  Tests: 
APT  vs.  ASVAB 


Ginger  Nelson  Goff 
Metrica,  Inc. 


Linda  Sawin 
Jim  Earles 

Air  Force  Research  Laboratory 
Brooks  Air  Force  Base 


Abstract 

The  latent  variable  structure  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  a  traditional,  paper-and- 
pencil,  multiple-aptitude  battery,  and  the  Advanced  Personnel  Testing  Battery  (APT),  a  computer  administered 
cognitive  abilities  battery,  were  examined  separately  and  together.  As  anticipated,  the  structure  of  the  ASVAB 
included  on  General  factor  and  three  or  four  specific  factors  depending  on  the  model  tested.  The  latent  variable 
structure  of  the  APT  conformed  to  expectations,  and  included  a  General  factor  and  three  orthogonal,  specific  factors. 
Results  of  the  joint  analyses  revealed  that  incorporating  tests  that  measure  fluid  intelligence,  spatial  ability,  and  fact 
learning  into  the  ASVAB  should  increase  or  improve  the  abilities  measured  by  that  instrument. 
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Introduction 


Multiple  aptitude  test  batteries  continue  to  be  of  interest  to  many  institutions  for  employee  placement  and 
selection  purposes,  including  govemiment  agencies  (Schmitt,  Sacked,  &  Cascio,  1996  and  Hedge,  Cartyer,  Borman, 
Morgan,  &  Foley,  1993)  and  private  businesses  (Hough-Dunette  &  Tippins,  1994).  A  potentially  new  direction  in 
individual  aptitude  testing  has  been  derived  from  cognitive  psychology  which  provides  the  theory  behind 
measurement  of  an  examinee’s  information  processing  capacity  instead  of  the  more  traditional  assessment  of  one’s 
knowledge  base  (Anderson,  1983,  Gustafsson  and  Muthen,  1994;  Kyllonen,  1993;  Kyllonen,  1994;  Kyllonen  & 
Alluisi,  1987,  Kyllonen  and  Christal,  1989). 

Within  the  above  context,  the  Advanced  Personnel  Testing  (APT)  tests  have  been  generated  and 
categorized  on  the  basis  of  a  consensus  model  of  information  processing.  This  is  a  subset  of  the  59-test  CAM4 
batteiy  (see  Kyllonen,  1994  for  a  detailed  description  of  the  theoretical  development  of  this  battery),  which  supports 
the  two  major  dimensions  of  (1)  cognitive  processes  and  (2)  content  domains.  The  cognitive  processes  dimension 
includes  six  domains:  working  memory  capacity  (WM),  declarative/fact  learning  ability  (FL),  procedural/skill 
learning  ability  (SL),  breadth  of  procedural  knowledge  (Induction;  IN),  breadth  of  declarative/general  knowledge 
(GK),  and  processing  speed  (PS).  Of  relevance  to  present  concerns  are  the  four  components  purported  to  measure 
processes  that  are  believed  to  be  potential  sources  of  individual  differences  in  cognitive  ability.  They  have  also  been 
related  to  success  in  various  learning  contexts  (Kyllonen  &  Christal,  1989).  These  components  can  be  defined  as 
follows.  Working  memory  capacity  measures  the  ability  to  simultaneously  store  old  information  and  to  process  new 
information.  Processing  speed  measures  the  ability  of  an  individual  to  retrieve  and  process  known  information. 
Declarative/fact  learning  taps  the  ability  of  the  individual  to  learn  new  facts,  while  procedural/skill  learning 
measures  the  ability  of  the  individual  to  learn  simple,  novel  rules  for  classifying  facts  (Kyllonen,  1993). 

The  content  dimension  consists  of  three  domains:  verbal,  quantitative  and  spatial.  The  three  content 
domains  are  hypothesized  to  reflect  individual  differences  in  relative  knowledge  (e.g.,  verbal  vs.  quantitative).  In 
turn,  these  domains  are  postulated  to  be  independent  of  general  differences  in  declarative  or  procedural  knowledge 
(Kyllonen,  1994).  Only  tests  representing  working  memory  capacity,  declarative/fact  learning,  procedural/skill 
learning,  and  induction  (which  tests  the  breadth  of  procedural  knowledge)  are  examined  in  this  study,  although  all 
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three  cognitive  domains  are  represented.  This  gives  us  a  four  by  three  matrix  generating  the  12  tests  employed  in 
the  present  investigation. 

This  study  examines  the  factor  structure  of  the  computer-administered  APT  battery  which  is  in  the  process 
of  being  validated.  The  structure  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) ,  a  traditional, 
multiple-choice  format,  multiple-aptitude  battery  used  by  all  the  services  to  select  and  classify  enlisted  personnel 
(Department  of  Defense,  1984),  is  also  presented.  In  addition,  the  APT  battery  was  factored  in  the  ‘presence’  of  the 
ASVAB. 

For  the  APT  Battery  a  nested  factor  model  (which  hypothesizes  one  general  factor  and  some  number  of 
orthogonal  specific  factors)  is  assumed.  Specific  factors  for  Fact  Learning,  Skill  Learning,  Induction,  Working 
Memory,  Quantitative,  Spatial,  and  Verbal  are  investigated  as  this  is  part  of  the  cognitive  theory  behind  the 
taxonomy  underlying  the  test  design.  It  is  hypothesized  that  the  factor  for  Working  Memory  will  be 
indistinguishable  from  the  General  factor  (Kyllonen,  1993, 1994).  Previous  confirmatory  factor  analyses  of  the 
current  version  of  the  ASVAB  subtests  have  reported  a  four  factor  structure  consisting  of  Verbal,  Speed,  Technical, 
and  Quantitative  factors  (Ree  &  Carretta,  1994;  Ree  &  Earles,  1990;  Ree  et  al.,  1982). 

Previous  studies  have  also  shown  that  enhancing  the  ASVAB  battery  with  tests  measuring  working 
memory,  reasoning,  and  spatial  visualization  produce  a  model  that  can  include  both  General  Intelligence  -  G,  and 
Crystallized  Intelligence  -  Gc  factors,  a  Broad  Visualization  (Gv)  or  Spatial  factor,  a  Speed  factor,  and  a 
Quantitative  factor  (Gustafsson  &  Muthen,  1994;  Sterling,  Goff,  &  Sawin,  1997).  Theoretically,  the  present  study 
assumes  that  our  combined  ASVAB-APT  data  set  has  a  very  similar  structure,  although  it  might  also  include  some 
of  the  specific  factors  hypothesized  in  the  APT  analyses. 

Method 

Participants 

Participants  were  6608  Air  Force  enlistees  tested  during  their  final  (sixth)  week  of  basic  training.  They 
were  considered  representative  of  USAF  enlisted  personnel  at  that  point  of  basic  training.  Data  was  gathered 
between  March  1994  and  July  1995.  Testing  was  conducted  in  mixed  sex  groups  of  no  more  than  40.  Before 
administering  the  battery  the  test  proctor  informed  all  individuals  that  their  participation  was  voluntary.  Participants 
were  also  assured  that,  in  accordance  with  the  Privacy  Act,  all  information  they  provided  would  be  strictly 
confidential. 
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Although  the  complete  sample  size  was  7088,  not  all  recruits  finished  every  test;  6669  recruits  had 
complete  data  on  all  12  of  the  APT  subtests  administered  in  the  first  set  and  also  had  valid  data  on  the  AFQT 
composite.  Because  participation  was  voluntary,  we  adopted  the  decision  rule  to  exclude  individuals  who  appeared 
to  have  under-performed  on  the  focal  set  of  12  APT  tests  given  their  AFQT  scores.  We  operationalized  this  rule  by 
regressing  total  scores  (correct/incorrect)  for  each  the  12  focal  APT  tests  on  AFQT  scores  and  then  computing 
standardized  residual  scores.  Unmotivated  participants  were  identified  as  those  who  obtained  (across  the  12 
subtests)  3  or  more  standardized  residuals  (z-scores)  that  were  less  than  or  equal  to  -3.0.  Using  this  procedure  we 
identified  61  participants  who  appeared  to  be  non-motivated  performers.  Accordingly,  we  eliminated  these 
individuals  from  the  data  set,  leaving  an  effective  sample  size  of  6608.  Of  the  6608  participants,  5367  (81.2%)  were 
males  and  1241  (18.8%)  were  female.  With  respect  to  race/ethnicity,  4988  (75.5%)  of  the  recruits  were  Caucasian, 
886  (13.4%)  were  African  American,  378  (5.7%)  were  Hispanic,  152  (2.3%)  were  Asian,  61  (.9%)  were  Native 
American,  and  143  (2.2%)  responded  they  were  Other.  The  modal  age  of  participants  was  18  (32%).  Sixty-one 
percent  of  subjects  had  completed  their  high  school  education,  while  33%  had  completed  some  college  hours. 
Examinees  had  an  average  AFQT  percentile  score  of  67. 

One  concern  with  this  data  is  that  the  recruits  represent  a  selective  sample  compared  to  the  entire 
population  who  take  the  ASVAB.  This  can  lead  to  problems  due  to  range  restriction,  such  as  lower  correlations 
between  measures  in  the  restricted  sample  compared  to  the  unrestricted  population.  Using  the  correlation  matrix, 
means,  and  standard  deviations  fi-om  the  ASVAB  1980  normative  sample  (Wegner  &  Ree,  1984;  Bock  &  Mislevy, 
1981),  the  correlations,  means,  and  standard  deviations  for  the  APT  battery  were  corrected  for  range  restriction. 
Analyses  were  performed  on  this  data  as  well  as  the  unrestricted  data.  Results  fi-om  the  corrected  analyses  did  not 
differ  much  from  the  original  analyses,  they  are  presented  in  Appendix  A. 


Instruments 

Participants  were  tested  on  Unisys  386  25  MHz  microcomputers  with  15"  multisync  non-interlaced  color 
monitors.  The  17  APT  subtests  were  administered  in  two  fixed  sets,  with  a  five  minute  break  between  sets.  There 
were  12  and  5  subtests  in  the  first  and  second  sets,  respectively.  Items  were  administered  randomly  within  each  test, 
and  tests  were  administered  randomly  within  each  set  with  a  few  constraints  on  the  subtest  ordering  such  as  the 
working  memory-spatial  and  fact  learning-spatial  tests  not  being  presented  consecutively.  We  will  use  the 
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information  from  the  first  12  of  the  17  tests  administered.  The  data  set  also  included  the  ten  test  scores  on  the 
ASVAB  Test  Battery  for  each,  as  well  as  the  usual  composite  scores.  The  battery  is  available  in  both  pencil-and- 
paper  and  computer-adaptive  (CAT-ASVAB)  forms.  Scores  on  both  forms  are  parallel  to  each  other  and  to  the 
reference  form  of  the  ASVAB.  The  scores  used  in  the  present  study  were  taken  from  the  tests  administered  to  each 
recruit  before  his/her  admittance  into  the  military.  Most  enlistees  were  tested  using  the  traditional  pencil-and-paper 
format. 

The  APT  tests  are  a  subset  of  the  CAM4  battery,  developed  by  researchers  at  the  Armstrong  Laboratories 
Human  Resource  Division  by  the  LAMP  team  (see  Kyllonen,  1994  for  a  detailed  account  of  the  theoretical 
development  of  tiie  CAM4  battery).  The  LAMP  tests  were  generated  and  categorized  on  the  basis  of  a  taxonomy 
based  on  the  consensus  model  of  information  processing,  as  well  as  research  which  suggests  that  the  content  of  the 
test  also  provides  systematic  variance  in  individual  test  scores.  Although  the  taxonomy  is  multidimensional,  the  two 
major  dimensions  which  drove  test  development  include  cognitive  processes  and  content  domains.  These 
dimensions  give  rise  to  the  theoretical  model  for  the  APT  tests  examined  in  this  study. 

The  cognitive  processes  dimension  includes  six  domains:  capacity  of  working  memory,  declarative/fact 
learning  ability,  procedural/skill  learning  ability,  breadth  of  procedural  knowledge  (Induction),  breadth  of 
declarative/general  knowledge,  and  processing  speed.  The  content  dimension  consists  of  three  domains:  verbal, 
quantitative  and  spatial.  The  three  content  domains  are  hypothesized  to  reflect  individual  differences  in  relative 
knowledge  (e.g.,  verbal  vs.  quantitative)  which  are  independent  of  the  general  differences  in  declarative  or 
procedural  knowledge  (Kyllonen,  1994).  Only  tests  representing  the  first  four  cognitive  processes  are  examined  in 
this  study,  but  all  three  cognitive  domains  are  represented.  This  gives  us  a  four  by  three  matrix  generating  the  12 
tests;  Working  Memory-Quantitative  (WMQ),  Working  Memory  Spatial  (WMS),  Working  Memory-Verbal 
(WMV),  Inductive  Reasoning-Quantitative  (INQ),  Inductive  Reasoning-Spatial  (INS),  Inductive  Reasoning-Verbal 
(INV),  Fact  Learning-Quantitative  (FLQ),  Fact  Learning-Spatial  (FLS),  Fact  Learning- Verbal  (FLV),  Skill 
Learning-Quantitative  (SLQ),  Skill  Learning-Spatial  (SLS),  Skill  Learning-Verbal  (SLV).  The  name  of  each  test 
(e.g.,  WMQ,  working  memory  quantitative)  indicates  the  cognitive  process  and  content  domain  being  measured. 
Detailed  descriptions  of  the  APT  tests  used  in  this  study  can  be  found  in  Appendix  B. 

The  traditional,  knowledge-based  ASVAB  battery  consists  of:  verbal  measures  of  general  science  (GS), 
word  knowledge  (WK),  and  paragraph  comprehension  (PC);  quantitative  measures  of  arithmetic  reasoning  (AR)  and 
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math  knowledge  (MK),  and  technical  measures  of  auto  and  shop  knowledge  (AS),  mechanical  comprehension  (MC), 
and  electronic  information  (El).  There  are  also  two  tests  that  measure  perceptual  speed:  numerical  operations  (NO) 
and  coding  speed  (CS).  Appendix  C  contains  descriptions  of  the  ASVAB  tests  used  in  this  study. 

Analyses 

A  series  of  confirmatory  factor  analyses  were  performed  using  the  STREAMS  shell  (Gustafsson  &  Stahl, 
1996)  which  interacts  with  LISREL  8.12  (Joreskog  &  Sorbom,  1993,  used  in  this  study),  as  well  as  EQS  (Bentler, 
1993).  Data  for  the  analyses  were  the  twelve,  standardized,  percent-correct  APT  scores.  As  described  above,  the 
analyses  include  a  number  of  models  based  on  the  nested  factor  model  which  hypothesizes  one  general  factor  and 
some  number  of  orthogonal  specific  factors.  Specific  factors  for  Fact  Learning,  Skill  Learning,  Induction,  Working 
Memory,  Quantitative,  Spatial,  and  Verbal  are  investigated.  It  was  hypothesized  that  the  factor  for  Working 
Memory  will  be  indistinguishable  from  the  general  factor  (Kyllonen,  1993;  Kyllonen,  1994),  but  our  original  theory 
assumed  that  the  rest  of  the  specific  factors  would  be  found. 

A  similar  set  of  analyses  were  done  on  the  ASVAB  scores.  Three  sets  of  scores  were  analyzed.  The  first 
set  was  comprised  of  all  ten  subtests.  Since  the  services  are  considering  dropping  some  of  the  tests  in  the  battery, 
the  second  set  excluded  Numbers  and  Operations  (NO),  while  the  third  excluded  NO  and  Coding  Speed  (CS).  For 
all  three  subsets  a  Nested  Factors  model  with  three  or  four  constructs  was  hypothesized:  Mathematics,  Technical, 
Verbal,  and  Speed  (where  appropriate.) 

An  additional  analysis  was  performed  in  order  to  compare  the  overlap  of  the  APT  and  ASVAB.  First,  the 
factors  from  the  nested  factors  approach  for  both  the  APT  and  the  ASVAB  were  correlated.  Factor  loadings  were 
fixed  to  the  values  arrived  at  in  the  previous  analyses,  and  the  factors  were  allowed  to  correlate  across  the  batteries, 
while  remaining  uncorrelated  within  a  battery.  A  slightly  different  model  was  investigated  which  allowed  only  the 
General  factors  from  APT  and  ASVAB  to  be  correlated,  while  assuming  the  specific  factor  variances  between  the 
tests  were  uncorrelated.  The  third  analysis  determined  the  fit  of  a  model  comprised  of  variables  from  both  batteries 
with  one  General  factor  and  specific  factors  corresponding  to  those  found  in  the  previous  analyses.  Finally,  a  model 
was  tested  that  contained  a  General  factor  and  various  specific  factors  suggested  by  previous  research,  which  were 
allowed  to  contain  variables  from  both  batteries. 
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Results 


APT  -  Nested  Factors 

Results  from  the  analyses  using  the  nested  factors  approach  on  the  12  percent-correct  scores  are  listed  in 
Table  2.  Several  models  with  varying  numbers  of  factors  were  analyzed.  The  first  model  posited  a  General  factor 
underlying  all  the  scores  with  no  other  factors  necessary.  Examination  of  the  fit  statistics  suggested  that  while  this 
model  may  be  acceptable,  modifications  of  this  very  simple  structure  may  provide  a  better  model.  The  Root  Mean 
Square  Error  of  Approximation  (RMSEA)  (Brown  &  Cudeck,  1993;  Steiger,  1990)  takes  into  account  both  the 
precision  of  the  fit  statistic  and  the  population  error  of  approximation.  Guidelines  for  evaluating  the  RMSEA 
statistic  are  that  it  should  fall  between  .03  and  .08,  with  .05  or  lower  being  the  target  value.  The  Goodness  of  Fit 
Index  (GFI)  and  the  Chi-square  statistic  are  more  traditional  indices  of  fit.  One  looks  for  values  of  .95  and  above  for 
the  GFI.  While  the  Chi-square  statistic  (x^)  is  not  as  useful  for  large  sample  sizes,  some  researchers  look  at  the 

value  divided  by  (n/1000)  when  the  sample  size  is  much  larger  than  1000.  For  a  discussion  of  these  and  many  other 
fit  indices,  see  Joreskog  &  Sorbom  (1993)  and  Gustafsson  &  Stahl  (1996).  In  the  current  study,  the  RMSEA  was 
.05 1  for  the  one  factor  model,  while  the  GFI  =  .97.  The  x^  value  (967,  degrees  of  freedom  -df  =  54)  is  high  even 

when  one  takes  into  account  the  large  sample  size  of  6608. 

Several  two  factor  models,  all  including  a  general  factor  and  one  specific  factor,  were  fitted  to  the  data  as 
well.  The  additional  factor  was  one  of  the  following:  Working  Memory,  Skill  Learning,  Fact  Learning,  Induction, 
Quantitative,  Spatial,  or  Verbal.  The  results  indicate  that  Working  Memory,  Skill  Learning,  and  Quantitative  were 
indistinguishable,  given  a  general  factor.  Results  do  suggest  that  Fact  Learning,  Induction,  Spatial,  and  Verbal  are 
present  although  the  Verbal  factor  is  quite  weak.  Evaluating  the  improvements  in  fit,  one  finds  that  adding  the 
Spatial  factor  as  a  second  factor  results  in  the  best  fitting  model.  The  fit  indices  for  this  model  were  x\50)  =  544, 
RMSEA  =  .039,  and  GFI  =  .99. 

For  theoretical  reasons,  models  including  a  general  factor  and  two  specific  were  also  examined.  Working 
Memory,  Skill  Learning,  and  Quantitative  were  not  examined  because  they  could  not  be  determined  previously. 
Induction,  Fact  Learning,  and  Verbal  were  investigated  as  additional  factors.  Results  for  these  models  show  that 
both  Induction  and  Fact  Learning  provided  improvements  to  the  fit.  At  this  stage,  however,  the  Verbal  factor 
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disappeared.  A  final  four  factor  model  with  General,  Spatial,  Fact  Learning,  and  Induction  showed  a  slight  further 
improvement  in  fit. 

In  order  to  decide  on  a  final  model,  theoretical  issues  as  well  as  parsimony  issues  arise.  Parsimony 
suggests  the  simplest  model  that  adequately  fits  the  data  is  the  one  to  choose,  so  either  a  one  or  two  factor  model 
should  suffice.  Theoretical  issues  suggest  that  all  four  factors  are  necessary  as  the  factor  variances  are  all  significant 
and  the  specific  factors  were  all  posited  from  the  beginning  of  the  study.  The  results  from  this  model  are  presented 
in  Table  3. 

ASVAB  -  Nested  Factors 

The  results  for  the  ASVAB  nested  factors  models  can  be  seen  in  Tables  4  through  8.  The  results  from  the 
complete  battery  are  presented  in  Tables  4  and  5.  Table  6  presents  data  from  the  set  that  does  not  contain  Numbers 
and  Operations  (NO).  Finally,  Tables  7  and  8  present  the  results  from  the  various  models  for  the  8  variable  subset 
(no  NO  and  no  Coding  Speed  -  CS).  For  all  sets  of  data  we  found  a  General  factor  plus  three  or  four  specific 
factors:  Verbal,  Math,  Technical,  and  Speed  (in  the  first  two  sets).  While  Verbal  loaded  on  GS,  WK,  and  PC  for  the 
first  models,  if  one  pursues  a  best  fitting  model,  the  results  show  that  GS  does  not  load  on  Verbal.  Theoretical  and 
goodness-of-fit  issues  suggest  that  the  models  with  Verbal  loading  on  GS,  Technical  loading  on  GS  and  AR,  and 
with  Speed  loading  on  PC,  AR,  and  MK  are  preferable,  so  subsequent  analyses  used  one  of  those  models.  For  the 
10  variable  data,  we  used  a  model  with  General  Intelligence  (all  variables).  Quantitative  (AR,  MK,  and  NO), 
Technical  (AS,  MC,  El,  GS,  and  AR),  and  Speed  (NO,  CS,  PC,  AR,  and  MK;  see  Table  5  for  these  results).  For  the 
8  variable  data  set  (Table  8),  Quantitative  has  only  AR  and  MK;  Technical  has  AS,  MC,  El,  GS,  and  AR;  and  Speed 
is  not  included.  Further  analyses  were  not  performed  with  the  9  variable  subset,  but  it  should  be  noted  that  including 
the  Speed  factor  improved  the  fit  greatly. 

ASVAB  &  APT  Analyses 

Table  9  presents  the  correlations  between  factors  from  the  nested  models  for  the  APT  and  the  ASVAB  test 
batteries.  The  first  matrix  of  correlations  represents  the  10  variable  ASVAB  subset  data,  while  the  second  matrix 
represents  the  8  variable  data  subset.  Both  correlation  matrices  show  that  the  APT  General  factor  correlates  most 
highly  with  the  ASVAB  General  and  Quantitative  factors  and  has  low  correlations  with  the  other  two  or  three 
ASVAB  factors.  Verbal,  Technical,  and  Speed.  The  Fact  Learning  factor  has  small  negative  correlations  with 
General,  Technical  and  Speed,  and  has  a  weak  positive  correlation  with  Verbal.  The  correlation  with  Quantitative  is 
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positive,  but  non-significant,  in  one  group  and  zero  in  the  other.  Induction  has  a  moderate  positive  correlation  with 
the  ASVAB  General  factor  while  all  other  correlations  are  low  and  positive  except  the  correlation  with  Speed  which 
is  low  and  negative.  The  APT  Spatial  factor  is  positively  correlated  with  the  ASVAB  General  factor.  It  also  has 
moderate  negative  correlations  with  Verbal  and  Quantitative  and  a  small  negative  correlation  with  the  ASVAB 
Speed  factor.  If  one  hypothesizes  a  model  that  constrains  the  intercorrelations  between  the  two  test  batteries  to  just 
one  correlation  between  the  two  General  factors,  one  obtains  a  correlation  of  0.75  for  the  10  variable  ASVAB  set 
and  a  correlation  of  0.78  for  the  8  variable  data  set. 

Tables  10  through  13  present  the  results  of  analyses  on  the  22  and  20  (no  NO  &  CS)  variables.  The  models 
at  first  hypothesize  one  General  Intelligence  factor  and  then  include  the  specific  factors  found  in  the  previous 
analyses  as  well  as  the  General  factor.  The  fit  for  the  model  with  one  factor  was  not  acceptable  for  either  data  set. 
The  RMSEA  values  were  0.105  and  0.102,  while  the  GFI  values  were  0.78  and  0.80.  Given  the  larger  sets  of 
variables,  the  Induction  factor  was  not  stable.  Most  analyses  that  included  Induction  would  not  converge  without 
fixing  the  factor  loadings  to  being  equal;  while  in  others,  the  factor  variance  was  non-significant.  Inclusion  of  this 
factor  also  did  not  improve  the  fit  of  the  model.  While  the  fit  indices  for  the  models  comprised  of  the  factors  for  the 
separate  batteries  are  not  unreasonable,  they  still  show  plenty  of  room  for  improvement  of  fit.  For  both  data  sets,  the 
RMSEA  was  somewhat  high,  but  within  the  accepted  range  at  0.061  for  the  22-variable  data  set  and  0.063  for  the 
20-variable  set.  The  GFI  values  were  lower  than  optimal  at  0.93  and  0.94,  respectively. 

The  preferred  model  has  a  structure  which  included  a  General  Intelligence  factor  -  G,  Crystallized 
Intelligence  -  Gc,  Broad  Visual  Perception  -  Gv,  Technical,  Speed  (NO  and  CS  are  included).  Quantitative,  and  Fact 
Learning  factors  (see  Figure  1).  In  the  final  analyses  for  both  sets  of  variables  the  tests  that  loaded  on  the  Technical 
factor  included  GS  and  AR  fi-om  the  ASVAB.  For  the  Quantitative  factor,  originally  AR,  MK,  and  NO  (if 
available),  only  the  tqjpropriate  Induction  and  Fact  Learning  tests  fi'om  the  APT  battery  loaded  positively  and 
significantly  on  this  factor.  The  loadings  for  the  FLQ  tests  were  quite  small,  however,  at  0.10  and  0.11  for  the  22  or 
20  variable  data  set  respectively.  The  three  technical  tests  (AS,  MC,  and  El),  as  well  as  GS  and  AR,  fi-om  the 
ASVAB  battery  also  loaded  on  Gv  which,  as  expected,  contained  all  four  Spatial  content  tests  from  the  APT  Battery. 
Crystallized  Intelligence  included  all  of  the  ASVAB  tests  except  NO  and  CS,  as  well  as  FLV  and  INV  from  the 
APT.  Note  that  the  loading  for  FLV  was  quite  small  at  0.09  or  0.08,  although  it  was  significant.  The  Fact  Learning 
factor  does  not  change  with  the  inclusion  of  the  ASVAB  tests.  The  three  Fact  Learning  tests  are  still  the  only  ones 
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that  load  on  that  factor.  The  next  factor  is  only  found  in  the  22  variable  data  set.  The  structure  for  the  Speed  factor 
is  similar  to  the  one  found  in  Gustafsson  and  Muthen  (1994),  but  the  loadings  for  AR,  MK,  WK,  and  PC  are 
relatively  smaller  while  CS,  with  a  standardized  loading  of  0.98,  is  almost  indistinguishable  from  the  factor.  The  fit 
indices  for  this  model  show  a  great  improvement  over  the  previous  models  with  an  RMSEA  of  0.037  and  a  GFI  of 
0.98  for  both  sets  of  variables. 

Figure  1 

.  Model  for  Table  11; 


Discussion 

TTie  theory  underlying  the  APT  test  battery  states  that  there  are  separate  cognitive  processes  and  separate 
content  domains:  Working  Memory,  Skill  Learning,  Fact  Learning,  and  Induction;  and  Quantitative,  Spatial,  and 
Verbal  respectively  (Kyllonen,  1993;  Kyllonen,  1994;  Kyllonen  &  Alluisi,  1987).  Confirmatory  Factor  Analysis  of 
the  available  data  set  support  only  aspects  of  this  theory.  Thus,  in  all  of  the  analyses  performed.  Working  Memory 
and  Skill  Learning  are  indistinguishable  from  each  other  and  from  General  Intelligence.  Separate  Fact  Learning, 
and  Induction  factors,  however,  are  found  in  these  analyses.  Of  the  three  content  factors  that  were  originally 
theorized,  only  the  Spatial  factor  is  supported  in  this  data  set. 
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One  can  speculate  as  to  the  reason  why  these  analyses  do  not  support  the  original  model.  Perhaps  the 
underlying  theory  is  incorrect  at  least  in  part,  but  that  conclusion  is  premature.  These  data  derive  from  a  restricted 
sample  of  the  population  and  even  correcting  for  range  restriction  may  not  adequately  address  this  problem.  The 
low  amount  of  explained  variance  for  each  test  suggests  that  there  may  be  problems  with  the  construction  of  the  test 
battery.  Since  there  is  only  one  test  for  each  process/content  area  in  this  battery  (e.g.  Induction-Verbal  or  Working 
Memory-Spatial),  it  is  impossible  to  distinguish  measurement  error  and  unique  test  variance  so  either  or  both  of  the 
next  suggestions  may  be  true.  The  specific  test  component  (e.g.  Fact  Learning  Quantitative)  may  override  any 
distinguishable  Quantitative  or  Verbal  factors.  Alternatively,  some  of  the  tests  may  have  poor  reliability  and  thus 
may  not  have  much  Verbal  or  Quantitative  variation  to  contribute  to  one  of  those  content  factors. 

The  structure  of  the  ASVAB  found  in  this  study  is  quite  similar  to  findings  in  previous  studies.  We  find  a 
General  factor  plus  three  or  four  specific  factors:  Verbal,  Quantitative,  Technical,  and  Speed  (where  applicable). 
While  some  of  the  studies  have  factors  which  consist  of  slightly  different  variable  sets,  all  agree  on  which  variables 
have  high  loadings  for  a  factor  and  thus  are  theoretically  the  most  important  to  the  factors.  For  instance,  some 
studies  define  the  Verbal  factor  to  include  just  Word  ICnowledge  and  Paragraph  Comprehension,  while  others  also 
include  General  Science  and/or  Electronic  Information  as  well.  One  should  note  that  the  inclusion  of  GS  and  AR  in 
the  Technical  factor  found  in  this  study  suggests  that  this  factor  may  include  science  or  technical  vocabulary 
competence  or,  perhaps,  spatial  ability. 

Unlike  the  ASVAB  analyses,  which  held  no  surprises,  the  findings  of  the  factor  correlations  and  joint 
analyses  of  the  APT  and  ASVAB  are  compelling.  The  important  conclusions  from  the  correlations  are  as  follows. 
The  APT  General  factor  correlates  most  highly  with  ASVAB  Quantitative  and  ASVAB  General,  which  suggests  that 
General  Intelligence  as  measured  by  the  APT  is  related  as  much  to  Crystallized  Intelligence  (Gc)  as  it  is  to  Fluid 
Intelligence  (Gf).  The  positive  correlation  between  the  APT  Spatial  and  ASVAB  General  factors  and  the  negative 
correlations  between  Spatial  and  the  ASVAB  Verbal  and  Quantitative  factors  suggest  that  the  ASVAB  General 
factor  contains  Broad  Visualization  (Gv)  components,  and  perhaps  Gf  (or  WM),  as  well  as  Gc,  since  Spatial  abilities 
are  usually  highly  correlated  with  the  Fluid  Intelligence  domain  (Carroll,  1993).  In  light  of  later  findings,  it  is 
interesting  to  note  that  Spatial  does  not  correlate  much  with  the  Technical  factor.  This  suggests  that  the  variance  of 
the  tests  underlying  this  factor  that  is  attributable  to  visual  processes  is  subsumed  by  the  General  factor  in  the 
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ASVAB  battery.  Also,  the  low  correlations  of  Fact  Learning  with  all  of  the  ASVAB  factors  suggest  that  this  process 
is  not  measured  at  all  by  the  current  ASVAB  tests. 

As  mentioned  previously,  Gustafsson  and  Muthen  (1994)  found  very  similar  results  when  the  ASVAB  is 
augmented  with  other  tests  that  measure  domains  not  well-represented  on  the  ASVAB.  Our  study  did  distinguish 
between  Gv  and  Technical  and  found  a  factor  for  the  Fact  Learning  process  as  well.  Note  also  that  only  the 
appropriate  Induction  and  Fact  Learning  tests  from  the  APT  battery  load  positively  and  significantly  on  the 
Quantitative  factor.  This  suggests  that  the  tests  in  this  content  domain  of  the  APT  should  be  re-evaluated  and 
perhaps  revised  if  they  are  to  reflect  the  Quantitative  content  area.  Also  of  interest  is  that  Gc  is  made  up  of  all 
ASVAB  tests  except  NO  and  CS,  as  well  as  FLV  and  INV  from  the  APT  battery.  One  would  expect  the  verbal  tests 
from  the  APT  battery  to  load  on  this  factor  and  perhaps  also  quantitative  tests  as  previous  studies  have  found, 
although  this  result  was  not  presently  obtained  (see  Carroll,  1993  for  an  overview  of  this  area  of  research).  The 
General  factor  is  weighted  towards  the  working  memory  and  skill  learning  tests  from  the  APT  battery  as  well  as 
some  of  the  quantitative  tests  from  the  ASVAB  battery.  This  finding,  coupled  with  the  existence  of  the  Gc  and  Gv 
factors  in  the  model,  validates  the  assumption  that  this  General  factor  may  be  interpreted  as  Fluid  Intelligence. 

Conclusions 

In  summary,  these  findings  support  previous  research  done  in  the  field  of  cognitive  theory  that  suggests  that 
G,  Gf,  and  WM  are  indistinguishable  from  each  other,  while  Gc  and  Gv  are  separate  constructs  (Gustafsson  & 
Muthen,  1994;  Kyllonen  &  Christal,  1990).  This  finding,  along  with  research  done  on  the  ECAT  (Alderton,  Wolfe, 
&  Larson,  1997;  Wolf,  Alderton,  Larson,  &,  Held,  1995)  and  on  other  experimental  batteries,  supports  the 
suggestion  of  incorporating  one  or  more  tests  that  measure  fluid  intelligence,  working  memory,  or  reasoning  into  the 
ASVAB  battery.  Another  important  finding  is  that  one  can  distinguish  Gv  from  the  Technical  factor  in  this  study. 
This  suggests  that  the  addition  of  one  or  two  purely  spatial  tests  to  the  ASVAB  battery  may  be  enough  to  permit  a 
similar  finding.  The  APT  battery  does  provide  an  additional  factor.  Fact  Learning,  that  is  not  found  in  the  ASVAB 
battery  at  all.  This  cognitive  domain  might  also  provide  some  new  tests  to  augment  the  ASVAB  battery. 
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Table  1 :  Means  and  Standard  Deviations 

N=6608 

APT: 

Working  Memory 

Quantitative 

50.00 

10.00 

Spatial 

50.00 

10.00 

Verbal 

Skill  learning 

50.00 

10.00 

Quantitative 

50.01 

10.00 

Spatial 

50.00 

10.00 

Verbal 

Induction 

50.00 

10.00 

Quantitative 

50.00 

10.00 

Spatial 

50.00 

10.00 

Verbal 

Fact  Learning 

50.00 

10.00 

Quantitative 

50.00 

10.00 

Spatial 

50.00 

10.00 

Verbal 

ASVAB: 

50.00 

10.00 

General  Science 

54.77 

6.62 

Arithmetic  Reasoning 

55.09 

6.60 

Word  Knowledge 

54.38 

4.33 

Paragraph  Comprehension 

54.81 

4.60 

Auto  Shop 

51.46 

8.20 

Math  Knowledge 

57.40 

6.50 

Mechanical  Comprehension 

55.27 

8.06 

Electronic  Information 

52.77 

7.69 
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Table  2:  APT  Battery  Structure 

Model  Fit  Statistics 


Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

1)  General  (G) 

54 

967 

.051 

.97 

2a)  G  +  Spatial  (Spat) 

50 

544 

.039 

.99 

2b)  G  +  Induction  (In) 

51 

903 

.050 

.98 

2c)  G  +  Fact  Learning  (FL) 

51 

909 

.050 

.98 

3a)  G  +  Spat  and  In 

47 

472 

.037 

.99 

3b)  G  +  Spat  and  FL 

47 

425 

.035 

.99 

3b)  G  +  In  and  FL 

48 

849 

.050 

.98 

4a)  G  +  Spat,  FL,  &  In 

44 

362 

.033 

.99 

Table  3:  Results  from  Final  APT  Model 

Includes  General  Intelligence, 

Spatial,  Fact  Learning,  and  Induction  Factors 


Goodness  of  Fit  Tests: 

Chi-square  =  361 .72,  df  =  44,  p  <  .00 

RMSEA  =  .033,  p-value  for  RMSEA  <  0.05  =  1.00 

Fit  Indices:  GFI  =  .99,  AGFI  -  .98,  NFI  =  .98,  NNFI  =  .97,  CFI  =  .98 

T-Values  for  Factor  Variances: 

General  =  24.5 1 
Spatial  =  9.69 
Fact  Learning  =  3 .99 
Induction  =  2.53 


Standardized  estimates: 


General 

Spatial 

Fact  Learning 

Induction 

Unique  Variance 

WMQ 

0.51 

0.74 

WMS 

0.62 

0.32 

0.51 

WMV 

0.56 

0.69 

SLQ 

0.60 

0.64 

SLS 

0.53 

0.23 

0.66 

SLV 

0.54 

0.71 

INQ 

0.45 

0.19 

0.76 

INS 

0.47 

0.19 

0.34 

0.63 

INV 

0.44 

0.16 

0.78 

FLQ 

0.39 

0.24 

0.79 

FLS 

0.57 

0.58 

0.22 

0.30 

FLV 

0.42 

0.30 

0.73 
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Table  4:  ASVAB  Battery 


10  Subtests  (include  NO  &  CS) 


Model  Fit  Statistics 


Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

1)  General  (G) 

35 

7762 

.183 

.79 

2)  G  +  Verbal,  Technical, 

25 

1172 

.083 

.96 

Quant,  &  Speed 

3)  G  +  Verbal,  Technical, 

21 

659 

.068 

.98 

Quant,  and  modified  Speed 

4)  G  +  Verbal,  modified  Tech, 

19 

500 

.062 

.99 

Quant,  and  modified  Speed 

5)  G  +  Verbal,  modified  Tech, 

18 

484 

.063 

.99 

Quant,  and  modified  Speed 

In  Model  2:  GS,  WK,  &  PC  tests  make  up  Verbal  factor. 

AS,  MC,  &  El  tests  make  up  Technical  factor. 

AR,  NO,  &  MK  tests  make  up  Quantitative  factor. 
NO  &  CS  make  up  Speed  factor. 

In  Model  3:  add  PC,  AR,  &  MK  to  Speed. 

In  Model  4:  add  GS  &  AR  to  Technical. 

In  Model  5:  add  WK  to  Speed. 
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Table  5:  Results  from  Final  ASVAB  Model 

(10  Variables) 

Includes  General  Intelligence, 

Technical,  Verbal,  Quantitative,  and  Speed  Factors 


Goodness  of  Fit  Test: 

Chi-square  =  499.67,  df  =  19,  p  <  .00 
RMSEA  =  .062,  p-value  for  RMSEA  <  0.05  =  .00 


Fit  Indices:  GFI  =  .99,  AGFI  =  .96,  NFI  =  .98,  NNFI  =  .95,  CFI  =  .98 

T-values  for  Factor  Variances: 

General  =  25.06 
Verbal  =  9.62 
Technical  =  14.08 
Quantitative  =  10.49 
Speed  =  10.37 

Standardized  estimates: 


General 

Verbal 

Technical 

Quantitative 

Speed 

Unique  Variance 

GS 

0.67 

0.27 

0.13 

0.46 

AR 

0.56 

0.13 

0.49 

0.20 

0.39 

WK 

0.51 

0.74 

0.19 

PC 

0.39 

0.31 

0.15 

0.73 

NO 

-0.02 

0.22 

0.65 

0.53 

CS 

-0.01 

0.88 

0.23 

AS 

0.37 

0.88 

0.09 

MK 

0.53 

0.52 

0.21 

0.40 

MC 

0.68 

0.36 

0.41 

El 

0.62 

0.41 

0.45 
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Table  6:  ASVAB  Battery  9  Subtests  (includes  CS,  not  NO) 


Model  Fit  Statistics 


Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

1)  General  (G) 

27 

4792 

.163 

.84 

2)  G  +  Verbal,  Technical, 

20 

1142 

.092 

.96 

and  Quant 

3)  G  +  Verbal,  Technical, 

16 

612 

.075 

.98 

Quant,  and  Speed 

4)  G  +  Verbal,  modified  Tech, 

14 

461 

.069 

.99 

Quant,  and  Speed 

5)  G  +  Verbal,  modified  Tech, 

13 

452 

.072 

.99 

Quant,  and  modified  Speed 


In  Model  2:  GS,  WK,  &  PC  tests  make  up  Verbal  factor. 

AS,  MC,  &  El  tests  make  up  Technical  factor. 
AR  &  MK  tests  make  up  Quantitative  factor. 

In  Model  3:  CS,  PC,  AR,  &  MK  make  up  Speed. 

In  Model  4:  add  GS  &  AR  to  Tech. 

In  Model  5:  add  WK  to  Speed. 
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Table  7:  ASVAB  Battery  8  Subtests  (does  not  include  NO  &  CS) 

Model  Fit  Statistics 


Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

General  (G) 

20 

4250 

.179 

.85 

G  +  Verbal,  Technical, 
and  Quant 

14 

848 

.095 

.97 

G  +  Verbal,  Technical, 
and  Quant 

13 

609 

.083 

.98 

G  +  Verbal,  modified 
Technical,  and  Quant 

11 

418 

.075 

.98 

In  Model  2:  WK  &  PC  tests  make  up  Verbal  factor. 

AS,  MC,  El,  &  GS  tests  make  up  Technical  factor 
(GS  has  a  stand,  loading  of  .06  on  Technical). 

AR  &  MK  tests  make  up  Quantitative  factor. 

In  Model  3:  GS,  WK,  &  PC  tests  make  up  Verbal  factor, 

AS,  MC,  &  El  tests  make  up  Technical  factor. 

AR  &  MK  tests  make  up  Quantitative  factor. 

In  Model  4:  GS,  WK,  &  PC  tests  make  up  Verbal  factor. 

AS,  MC,  El,  AR  &  GS  tests  make  up  Technical  factor. 
AR  &  MK  tests  make  up  Quantitative  factor. 


20 


Table  8:  Results  from  Final  ASVAB  Model  (8  Variables) 

Includes  General  Intelligence, 

Technical,  Verbal,  and  Quantitative  Factors 

Goodness  of  Fit  Test: 

Chi-square  =  418.19,  df-  1 1,  p  <  00 

RMSEA  =  .075,  p-value  for  RMSEA  <  0.05  =  .00 

Fit  Indices:  GFI  =  .98,  AGFI  =  .95,  NFI  -  .98,  NNFI  =  .94,  CFI  =  .98 

T-values  for  factor  variances: 


General  =  20.36 
Verbal  =  8.00 
Technical  =  17.32 
Quantitative  =  24. 1 5 

Standardized  estimates: 


General 

Verbal 

Technical 

Quantitative 

Unique  Variance 

GS 

0.67 

0.25 

0.16 

0.46 

AR 

0.57 

0.13 

0.53 

0.38 

WK 

0.51 

0.78 

0.13 

PC 

0.42 

0.28 

0.75 

AS 

0.33 

0.87 

0.14 

MK 

0.55 

0.54 

0.41 

MC 

0.65 

0.40 

0.41 

El 

0.59 

0.45 

0.45 
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Table  9: 


APT 

General 
Spatial 
Fact  Learn 
Induction 
ASVAB 
General 
Verbal 
Technical 
Quantitative 
Speed 


APT 

General 
Spatial 
Fact  Learn 
Induction 
ASVAB 
General 
Verbal 
Technical 
Quantitative 


Correlations  between  APT  Factors  and  ASVAB  Factors 

(10  Variables  for  ASVAB) 


APT 


ASVAB 


Gen 

Spat 

FactLm 

Indue 

Gen 

Verb 

Tech 

Quant 

Speed 

1.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.00 

0.00 

0.00 

1.00 

0.55 

0.40 

-0.11 

0.38 

1.00 

0.06 

-0.38 

0.16 

0.09 

0.00 

1.00 

-0.01 

0.09 

-0.09 

0.06 

0.00 

0.00 

1.00 

0.47 

-0.27 

0.07 

0.14 

0.00 

0.00 

0.00 

1.00 

0.24 

-0.12 

-0.11 

-0.10 

0.00 

0.00 

0.00 

0.00 

1.00 

Correlations  between 

APT  Factors  and  ASVAB  Factors 

(8  Variables  for  ASVAB) 

APT 

ASVAB 

Gen 

Spat 

FactLm 

Indue 

Gen 

Verb 

Tech 

Quant 

1.00 

0.00 

1.00 

0.00 

0.00 

1.00 

0.00 

0.00 

0.00 

1.00 

0.55 

0.37 

-0.09 

0.38 

1.00 

0.07 

-0.35 

0.13 

0.07 

0.00 

1.00 

-0.02 

0.15 

-0.11 

0.08 

0.00 

0.00 

1.00 

0.55 

-0.29 

0.00 

0.07 

0.00 

0.00 

0.00 

1.00 
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Table  10:  ASVAB  and  APT  Batteries~10  ASVAB  Tests 


Note:  includes  NO  &  CS 

Model  Fit  Statistics 


Degrees  of  Freedom 

1)  General  (G)  209 

2)  G  +  Verbal,  Spatial,  Speed,  192 

Technical,  Quant,  &  Fact 
Learning 

3)  G  +  Verbal,  Spatial,  Speed,  191 
Technical,  Quant,  Fact 
Learning,  &  Induction 

4)  G  +  Gc,  Gv,  Speed,  171 

Technical,  Quant,  & 

Fact  Learning 


Chi-Square 

RMSEA 

GFI 

15382 

.105 

.78 

4967 

.061 

.93 

4959 

.061 

.93 

1748 

.037 

.98 

In  Model  2:  GS,  WK,  &  PC  tests  make  up  Verbal. 

FLS,  SLS,  INS,  &  WMS  make  up  Spatial. 

NO  &  CS  make  up  Speed. 

AS,  MC,  &  El  tests  make  up  Technical. 

AR,  NO,  &  MK  tests  make  up  Quantitative. 

FLQ,  FLS,  &  FLV  make  up  Fact  Learning. 

In  Model  3:  INQ,  INS  &  INV  make  up  Induction  (In). 

Loadings  for  In  all  fixed  to  1  so  model  will  converge. 

In  Model  4:  Model  does  not  have  Induction. 

Verbal  becomes  Crystallized  Intelligence  (Gc). 

It  is  made  up  of  all  ASVAB  except  NO  &  CS 
as  well  as  IW  and  FLV  from  the  APT  Battery. 
Spatial  becomes  Broad  Visual  Perception  (Gv). 

It  is  made  up  of  FLS,  SLS,  INS,  and  FLS  from  APT 
as  well  as  AS,  MC,  El,  GS,  and  AR  from  ASVAB. 
Add  PC,  AR,  MK  to  Speed. 

Add  GS  &  AR  to  Teclmical. 

Add  INQ  and  FLQ  to  Quantitative. 
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Table  11:  Results  from  Final  APT  and  ASVAB  Model  (22  Variables) 

Includes  General  Intelligence, 

Crystallized  Intelligence,  Broad  Visual  Perception, 

Speed,  Technical,  Quantitative,  and  Fact  Learning  Factors 


Goodness  of  Fit  Test: 


Chi-square  =  1748.50,  df  =  172,  p  <  .00 

RMSEA  -  .037,  p-value  for  RMSEA  <  0.05  -  1.00 

Fit  Indices:  GFI  =  .98,  AGFI  =  .96,  NFI  =  .96,  NNFI  =  .95,  CFI  =  .96 

T-values  for  the  Factor  Variances: 


General  =  25.12 
Gc  =  23.91 
Gv=  13.67 
Speed  =  24.53 
Technical  =  19.55 
Quantitative  =  12.  63 
Fact  Learning  =  5.23 

Standardized  estimates: 


General 

Gc 

Gv 

Speed 

GS 

0.35 

0.65 

0.15 

AR 

0.60 

0.21 

0.09 

0.08 

WK 

0.28 

0.70 

0.04 

PC 

0.29 

0.46 

0.12 

NO 

0.14 

0.55 

CS 

0.18 

0.98 

AS 

0.12 

0.32 

0.28 

MK 

0.58 

0.19 

0.08 

MC 

0.41 

0.33 

0.43 

El 

0.23 

0.46 

0.28 

WMQ 

0.51 

WMS 

0.62 

0.33 

WMV 

0.54 

SLQ 

0.60 

SLS 

0.54 

0.27 

SLV 

0.53 

INQ 

0.46 

INS 

0.49 

0.23 

INV 

0.45 

0.30 

FLQ 

0.38 

FLS 

0.57 

0.47 

FLV 

0.40 

0.09 

Tech 

Quant  Fact  Learn 

Unique  Var 

0.14 

0.42 

0.20 

0.44 

0.35 

0.43 

0.69 

0.27 

0.60 

0.00 

0.68 

0.34 

0.42 

0.44 

0.43 

0.36 

0.49 

0.42 

0.74 

0.50 

0.71 

0.64 

0.64 

0.72 


0.21 

0.74 

0.71 

0.71 

0.11 

0.31 

0.75 

0.25 

0.39 

0.31 

0.74 
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Table  12:  ASVAB  and  APT  Batteries--8  ASVAB  Tests 


Note:  does  not  include  NO  &  CS 
Model  Fit  Statistics 


Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

1) 

General  (G) 

170 

11877 

.102 

.80 

2) 

G  +  Verbal,  Spatial,  Speed, 
Technical,  Quant,  &  Fact 
Learning 

156 

4290 

.063 

.94 

3) 

G  +  Verbal,  Spatial,  Speed, 
Technical,  Quant,  Fact 
Learning,  &  Induction 

155 

4283 

.063 

.94 

4) 

G  +  Gc,  Gv,  Speed, 

139 

1400 

.037 

.98 

Technical,  Quant,  & 

Fact  Learning 

In  Model  2:  GS,  WK,  &  PC  tests  make  up  Verbal. 

FLS,  SLS,  INS,  &  WMS  make  up  Spatial. 

AS,  MC,  &  El  tests  make  up  Technical. 

AR  &  MK  tests  make  up  Quantitative. 

FLQ,  FLS,  &  FLV  make  up  Fact  Learning. 

In  Model  3:  INQ,  INS  &  INV  make  up  Induction, 

loadings  all  fixed  to  1  so  model  will  converge. 

In  Model  4:  Model  does  not  have  Induction. 

Verbal  becomes  Crystallized  Intelligence  (Gc). 

It  is  made  up  of  all  ASVAB  except  NO  &  CS 
as  well  as  INV  and  FLV  from  the  APT  Battery. 
Spatial  becomes  Broad  Visual  Perception  (Gv). 

It  is  made  up  of  FLS,  SLS,  INS,  and  FLS  from  APT 
as  well  as  AS,  MC,  El,  GS,  and  AR  from  ASVAB. 
Add  GS  &  AR  to  Technical. 

Add  INQ  and  FLQ  to  Quantitative. 
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Table  13:  Results  from  Final  APT  and  ASVAB  Model  (20  Variables) 

Includes  General  Intelligence, 

Crystallized  Intelligence,  Broad  Visual  Perception, 

Technical,  Quantitative,  and  Fact  Learning  Factors 


Goodness  of  Fit  Test: 


Chi-square  =  1400.20,  df  =  139,  p  <  .00 


RMSEA  =  .037,  p-value  for  RMSEA  <  0.05  =  1.00 

Fit  Indices:  GFI  =  .98,  AGFI  =  .97,  NFI  =  .96,  NNFI  -  .96,  CFI  =  .97 

T-values  for  the  Factor  Variances: 

General  =  25.13 
Gc  =  25.04 
Gv=  13.71 
Technical  =  20.22 
Quantitative  =  8.71 
Fact  Learning  =  5.24 

Standardized  estimates: 


General 

Gc 

Gv 

Tech 

Quant 

Fact  Learn 

Unique  Var 

GS 

0.35 

0.64 

0.16 

0.16 

0.42 

AR 

0.60 

0.19 

0.07 

0.19 

0.48 

0.32 

WK 

0.28 

0.71 

0.42 

PC 

0.29 

0.45 

0.71 

AS 

0.13 

0.30 

0.28 

0.69 

0.34 

MK 

0.59 

0.18 

0.40 

0.46 

MC 

0.42 

0.31 

0.42 

0.44 

0.36 

El 

0.23 

0.45 

0.28 

0.50 

0.42 

WMQ 

0.51 

0.74 

WMS 

0.62 

0.33 

0.50 

WMV 

0.54 

0.71 

SLQ 

0.60 

0.64 

SLS 

0.54 

0.26 

0.64 

SLV 

0.53 

0.72 

INQ 

0.46 

0.21 

0.74 

INS 

0.49 

0.22 

0.71 

INV 

0.45 

0.30 

0.71 

FLQ 

0.37 

0.10 

0.31 

0.76 

FLS 

0.57 

0.48 

0.25 

0.38 

FLV 

0.40 

0.08 

0.31 

0.74 
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Appendix  A:  Selected  Analyses  with  Data  Corrected  for  Range 
Restriction 


Table  A 1 

APT  Battery  Structure 


Model 

Fit  Statistics 

Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

1)  General  (G) 

54 

1042 

.053 

.97 

2)  G  +  Spat,  FL,  &  In 

44 

368 

.033 

.99 

Table  A2 
ASVAB  Battery 
10  Subtests  (include  NO  &  CS) 

Model  Fit  Statistics 


Degrees  of  Freedom 

Chi-Square 

RMSEA 

GFI 

1)  General  (G) 

35 

9083 

.198 

.75 

2)  G  +  Verbal,  Technical, 

25 

1483 

.094 

.96 

Quant,  and  Speed 

3)  G  +  Verbal,  Technical, 

21 

942 

.081 

.97 

Quant,  and  modified  Speed 

4)  G  +  Verbal,  Technical, 

20 

880 

.081 

.98 

Quant,  and  modified  Speed 

5)  G  +  Verbal,  modified  Tech, 

18 

705 

.076 

.98 

Quant,  and  modified  Speed 


In  Model  2:  GS,  WK,  &  PC  tests  make  up  Verbal  factor. 

AS,  MC,  &  El  tests  make  up  Technical  factor. 

AR,  NO,  &  MK  tests  make  up  Quantitative  factor. 
NO  &  CS  make  up  Speed  factor. 

In  Model  3:  add  PC,  AR,  &  MK  to  Speed. 

In  Model  4;  add  WK  to  Speed. 

In  Model  5:  add  GS  and  AR  to  Technical. 
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Table  A3 

Correlations  between 

APT  Nested  Factors  and  ASVAB  Nested  Factors 
corrected  for  range  restriction 


APT  ASVAB 


Gen 

Spat 

FactLm 

Indue 

Gen 

Verb 

Quant 

Speed 

APT 

General 

1.00 

Induction 

0.00 

1.00 

Spatial 

0.00 

0.00 

1.00 

Fact  Learn 

0.00 

0.00 

0.00 

1.00 

ASVAB 

General 

0.57 

0.06 

0.07 

0.40 

1.00 

Verbal 

0.16 

-0.35 

0.04 

0.01 

0.00 

1.00 

Technical 

0.04 

0.34 

-0.26 

0.05 

0.00 

0.00 

1.00 

Quantitative 

0.56 

0.14 

-0.11 

0.09 

0.00 

0.00 

0.00 

1.00 

Speed 

0.24 

-0.19 

-0.11 

-0.11 

0.00 

0.00 

0.00 

0.00 
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Table  A4 

ASVAB  and  APT  Batteries 
10  ASVAB  Tests  (includes  NO  &  CS) 

Model  Fit  Statistics 

Degrees  of  Freedom  Chi-Square 

1)  General  (G)  209  15382 

2)  G  +  Verbal,  Spatial,  Speed,  185  4310 

Technical,  Quant,  &  Fact 
Learning 

3)  G  +  Verbal,  Spatial,  Speed,  184  4305  .058  .94 

Technical,  Quant,  Fact 

Learning,  &  Induction 

4)  G  +  Gc,  Gv,  Speed,  171  1826  .038  .97 

Technical,  (^ant,  & 

Fact  Learning 

In  Model  2:  GS,  WK,  &  PC  tests  make  up  Verbal. 

FLS,  SLS,  INS,  &  WMS  make  up  Spatial. 

NO,  CS,  AR,  WK,  PC,  &  MK  make  up  Speed. 

AS,  MC,  El,  GS,  &  AR  tests  make  up  Technical. 

AR,  NO,  &  MK  tests  make  up  Quantitative. 

FLQ,  FLS,  &  FLV  make  up  Fact  Learning. 

In  Model  3:  INQ,  INS  &  INV  make  up  Induction  (In), 

loadings  for  In  all  fixed  to  1  so  model  will  converge. 

In  Model  4:  Model  does  not  have  Induction. 

Verbal  becomes  Crystallized  Intelligence  (Gc). 

It  is  made  up  of  all  ASVAB  except  NO  &  CS 
as  well  as  INV  and  FLV  fi'om  the  APT  Battery. 

Spatial  becomes  Broad  Visual  Perception  (Gv). 

It  is  made  up  of  FLS,  SLS,  INS,  and  FLS  from  APT 
as  well  as  AS,  MC,  El,  GS,  and  AR  from  ASVAB. 

Add  INQ  and  FLQ  to  Quantitative. 


RMSEA 

GFI 

.105 

.78 

.058 

.94 
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Appendix  B:  Descriptions  of  the  APT  Tests 


Working  Memory: 

Quantitative  (WMQ)  10  minus  n  : 

Subjects  must  remember  the  last  three  digits  of  a  three  to  six  digit  string.  Each  digit  is  presented 
on  a  separate  screen.  If  the  digit  is  white,  the  subject  remembers  that  number.  If  it  is  red,  the 
subject  must  remember  ten  minus  that  number. 

Spatial  (WMS)  Synthesis  +/-  : 

Subjects  were  shown  two  successive  screens  with  matrices  of  dots,  some  of  which  were  connected 
by  lines.  Then  the  subject  was  asked  to  add  or  subtract  the  two  and  responded  by  connecting  the 
appropriate  dots  in  a  new  matrix. 

Verbal  (WMV)  Furniture-animals  order : 

Subjects  processed  three  successively  presented  screens  with  sentences  that  constrained  the  order 
of  a  set  of  four  items.  This  study  used  a  set  of  items  containing  two  animals  and  two  pieces  offumiture.  In 
one  sentence  the  order  for  the  animals  is  given,  in  another  the  pieces  of  furniture  are  grouped,  and  the  third 
gives  the  order  for  the  two  groups  (animals  and  furniture). 

Inductive  Reasoning: 

Quantitative  (INQ)  Number  matrices: 

Subjects  were  shown  a  3x3  matrix  of  numbers  with  one  missing  number.  Subjects  had  to  figure 
out  the  rules  governing  the  series  of  numbers  and  fill  in  the  missing  space. 

Spatial  (INS)  Figure  series: 

A  series  of  three  figures  was  presented  and  subjects  had  to  identify  the  next  figure  in  that  series. 

Verbal  (INV)  Word  sets: 

Subjects  are  shown  three  sets  of  words,  names,  or  phrases  and  must  identify  the  one  set  that  does 
not  belong  with  the  other  two  sets. 

Fact  Learning: 

Quantitative  (FLQ)  2-digit  blocks: 

A  block  of  numbers  was  presented  to  subjects  in  one  screen.  On  the  next  screen,  subjects  were 
asked  if  a  number  was  on  the  previous  list. 

Spatial  (FLS)  Palmer-Fig  Pairs: 

Two  figures  are  presented  separately  on  a  3x3  grid  of  dots.  Subjects  are  then  shown  one  of  the 
figures  and  are  asked  to  remember  the  other  one. 

Verbal  (FLV)  Noun-Pair  Lookup: 

Subjects  are  shown  8  pairs  of  words  to  memorize.  Then  a  pair  of  words  is  presented  and  the 
subject  must  decide  if  the  pair  is  in  the  list.  Practice  was  given  with  the  list  displayed  at  the  top 
of  the  screen.  Then  items  are  given  without  the  prompt  list. 


30 


Descriptions  of  the  APT  Tests 
(continued) 


Skill  Learning: 

Quantitative  (SLQ)  Odd-Big: 

Subjects  memorize  a  set  of  rules  governing  responses.  A  number  is  presented  and  the  subject 
must  decide  if  it  is  odd  or  even  and  if  it  is  big  or  little.  Then  the  subject  uses  the  rules  to  respond. 

Spatial  (SLS)  4-square: 

A  2x2  grid  of  blocks  is  displayed  and  series  of  four  dots  is  displayed,  one  in  each  block,  to  trace 
out  one  of  three  patterns.  The  subject  must  learn  the  patterns  and  then  identify  the  pattern  when 
series  are  presented  where  either  the  second  or  third  dot  fails  to  light  up. 

Verbal  (SLV)  Future-Past-Present: 

Two  words  appear  on  the  screen.  Subject  must  determine  if  they  represent  the  same  or  different 
time  periods.  Tlien  the  subject  is  presented  with  words  in  the  3  different  time  periods.  If  the 
original  pair  are  from  the  same  time,  the  subject  should  pick  the  same  time.  If  they  are  different, 
he  should  pick  the  word  representing  the  third  time  period. 
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Appendix  C:  Descriptions  of  the  ASVAB  Tests 


General  Science.  This  test  contains  35  standard  vocabulary  items,  such  as  “Which  of  the  following  foods  contain 
the  most  iron?  (a)  eggs  (b)  liver  (c)  candy  (d)  cucumber.” 

Arithmetic  Reasoning.  This  test  consists  of  30  arithmetic  word  problems. 

Word  Knowledge.  This  test  consists  of  35  standard  vocabulary  items,  such  as  “The  wind  is  variable  today,  (a)  mild 
(b)  steady  (c)  shifting  (d)  chilling.” 

Paragraph  Comprehension.  This  test  presents  15  paragraphs,  each  1-3  sentences  long,  followed  by  a  multiple- 
choice  response  question  about  the  paragraph’s  content. 

Numerical  Operations.  A  10-minute,  speeded  test,  this  consists  of  50  number-fact  items  (e.g  2X6=*?  (a)  4  lb)  8  fcl 
3  (d)  12. 

Coding  Speed.  A  10-minute,  speeded  test,  this  consists  of  84  items  designed  to  measure  how  quickly  one  can  find  a 
number  in  a  table. 

Auto  and  Shop  Information  (Autoshop).  This  test  consists  of  25  questions  about  automobiles,  shop  practices,  and  the 
use  of  tools. 

Mathematics  Knowledge.  This  test  consists  of  25  arithmetic  word  problems  (primarily  algebra,  but  also  simple 
geometry. 

Mechanical  Comprehension.  This  test  consists  of  25  questions,  normally  accompanied  by  drawings,  relating  to 
general  mechanical  and  physical  principles. 

Electrical  Information.  This  test  consists  of  20  questions  relating  to  electrical,  radio,  and  electronics  information. 
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Section  2:  APT  Briefing 


ADVANCED  PERSONNEL  TESTING 

Presented  To:  OASD  (FMP)  (MPP)  AP 

15  August  1996 


Presented  by: 

Dr  Linda  Sawin 
Cognitive  Technologies  Branch 
Armstrong  Laboratory,  Human  Resources  Directorate 
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"  Background 


"  USAF  Validation  Project 

"  Interim  Results 
Validity 
Fairness 
Classification 


Summary  and  Recommendation 


Background 


"  Historical  Influences 
I  Navy-ONR 
1  Army  -  Project  A 
I  DoD  -  CAT-ASVAB;  ECAT 
I  Air  Force  -  LAMP 

"  Current  Battery 
I  Theory  Driven 
I  Computer  -  Administered 


Overview 


"  Background 

=>  "  USAF  Validation  Project 


Ini 


:erim  Results 
Validity 
Fairness 
Classification 


Summary  and  Recommendation 


USAF 

Validation  Project 


"  Purpose:  To  Examine  the  Utility  of  APT  Information 
Processing  Battery  for  Selection  and 
Classification 

"  15  Schools  plus  2  Core  Courses 
High  Flow 

Range  of  Job  Types  and  Ability  Levels 
Large  Number  of  Females  and  Minorities 

"  Subjects:  15,000-20,000  Airman  Basics  Tested  at 
Lackland  AFB 

"  2-5  Years 


AFSC 

Career  Field 

Selector  Al 

1C131Y 

Air  Traffic  Control  Operator 

1N335A 

Arabic  Crypotologic  Linguist 

G-69 

1N431Y 

Signals  Intelligence  Analyst 

G-58 

2A333B 

F-16  Aircraft  Maintenance 

M-51 

2A431Y 

Aircraft  Guidance  and  Control 

E-S7 

2A531G 

Strategic  Aircraft  Maintenance  (K0135} 

M-51 

2A635Y 

Aircraft  Pneudraulic  Systems 

M-57 

2E031Y 

Ground  Radar  Systems 

E-67 

2E131Y 

Satellite  &  Wideband  Communications 

E-67 

2E133Y 

Ground  Radio  Communications 

E-67 

3A031Y 

Information  Management 

A-32 

3C231Y 

Communication  Computer  Systems  Control 

E-67 

3P031Y 

Security  Police 

G-35 

3S031Y 

Personnel  Specialist 

A-45 

4A031Y 

Medical  Administrative 

G-43 

Fixed  Wing  Aircraft 

M-51 

Electronics  Principles 

E-67+ 

Project  Status 


"  Over  9,000  Airman  Basics  Tested 

"  Completed  Validity  Data  Collection 
I  Security  Police 
I  Fixed  Wing  Aircraft  Mechanics 
I  Electronic  Principles 

"  Completed  Equity  Data  Collection:  Security  Police 


"  Background 


Overview 


"  USAF  Validation  Project 


In 


:erim  Results 
Validity 
Fairness 
Classification 


Summary  and  Recommendation 


Technical  Schools 


"  Security  Police 
I  Gen-35 
I  N  =  1,126 

ocM  =  950:F  =  176 
oc  W  =  840;  B  =  164;  Other=122 
"  Fixed  Wing  Aircraft 
Mech-51 
N  =  1,107 

ocM  =  1,080;  F  =  27 
oc  W  =  930;  B  =  80;  Other=97 
"  Electronic  Principles 
I  Elect-67-i- 
1  N  =726 

oc  M  =  663;  F  =  73 

oc  W  =  587;  B  =  66;  Other=73 


"  Validity 

Incremental 

Simple 

Constraints  on  Validity 
"  Fairness 

Differential  Item  Functioning  (DIF) 
Mean  Differences 
Predictive  Equity 

"  Classification  Efficiency 
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Results  Overview 


"  Validity 

I  Overali,  APT  Composites  are  incrementally  Valid 
over  ASVAB 

I  Tests  are  Valid  Predictors  of  Technical  School 
Performance 

"  Fairness:  APT  Exhibits  Smaller  Sex  and  Race 
Differences 

"  Ciassification  Utiiity:  Shows  Promise 


Composite  Incremental 
Validity  Analyses 

"  Criterion:  Finai  Schooi  Grade 

"  Predictors:  Average  of  Unit-Weighted  Subtests 
I  ASVAB-10 
I  APT-12 

I  APT  Process  Composites 

ocWorking  Memory  (WMS+WMV+WMQ) 
ocFact  Learning  (FLS+FLV+FLQ) 
ocSkili  Learning  (SLS+SLV+SLQ) 
oc Induction  (iNS+iNV+INQ) 


Correlations  are  unconnected 

A$VAB-10  and  APT-1 2  formed  by  averaging  unit-weighted  subtests 


Table  2 


Constraints  on  Validity 
Subtest  Difficulty 


n=6606 

Difficulty  defined  as  percent  correct;  number  of  response  options  in  parentheses 


Constraints  on  Validity 
Item  Response  Theory 

"  Discrimination  (a-parameter) 

Subtests  Show  Acceptable  Discrimination 
Discrimination  Could  Be  Increased 


ff 


Di 


fficulty  (b-parameter) 
Overall,  Tests  are  Easy 
One  Test  is  Difficult:  FLS 


"  Guessing  (c-parameter) 

I  Guessing  Correctly  Difficult 


Tables  3-14 


FLSU) 
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Fairness 

Mean  Differences  -  Race 


APT 


Fairness 

Predictive  Equity 

"  Tested  for  Differences 

Standard  Errors  of  Estimate  (SEE) 

Slopes 

Intercepts 


Results:  Security  Police 

No  Slope  or  SEE  Differences 
Overprediction  for  Females  and  Blacks 


Classification 
APT  and  ASVAB 
Intercorrelations 


"  APT  less  Intercorrelated 

I  Average  NORC  Corrected  Correlations 
ocASVAB-10  =  .71 
ocASVAB-8  =  .80 
(omit  NO  and  CS) 
ocAPT-12  =  .62 


"  Potential  for  Greater  Classification  Efficiency 


Correlations  corrected  for  unreliability 


Tables  20  and  21 


Overview 


"  Background 

"  USAF  Validation  Project 

"  Interim  Results 
Validity 
Fairness 
Classification 


=>  "  Summary  and  Recommendation 


Summary 

"  Validity 

I  Demonstrated  Incremental  Validity 
I  Validity  Could  Be  Improved  by  Increasing: 
ocitem  Difficulty 
ocitem  Discrimination 
ocTest  Reliability 
"  Fairness 

I  Smaller  Mean  Differences 
I  Predictive  Equity  Similar  to  ASVAB 


"  Potential  for  Classification  Efficiency:  Less  Intercorrelated 


Next  Steps 

"  Battery  Polishing  Underway 
I  Construct  More  Difficuit  Items 
I  Pilot  Test  More  Items  on  Airman  Basics 
I  Analyze  Results 

"  Deliver  Improved  Battery  to  Other  Services 


BACKUP 

SLIDES 
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APT  Composite  Incremental 
Validity  Over  AFQT 


Security  Police 
G-35 

(n  =  1,126) 


Fixed  Wing 
Aircraft  Mechanic 
M-51 

(n  =  1,107) 


Electronic 

Principles 

E-67+ 

(n  =  726) 


Change  in  uncorrected  multiple  correlation  (AR) 
Composites  are  unit'Weighted  and  averaged 


Fact 

Learning 


Skill 

Learning 


Working 

Memory 

Induction 

.008 

.007 

.002 

.002 

.036 

.022 

DIF  Indices 


"  Mantel  -  Haenszel  Chi-Square 
"  Mantel  -  Haenszel  Odds  Ratio/Delta  Difference 
"  Standardized  P-Difference 
"  Root-Mean-Weighted-Squared-Difference 
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Personnel 


APT  Scientists 

Jim  Earles 
Brad  Tiegs 
Meredith  Sterling 
Linda  Sawin 

APT  Support 

Mary  Becerra 
Terrilee  Perdue 
Joanne  Hall 
Wayne  Crone 


AL/HRM 

Melody  Darby 
Malcolm  Ree 
Lonnie  Valentine 
Jacobina  Skinner 
Ginger  Goff 


External  Scientists 

Fritz  Drasgow 

University  of  Illinois 
Mary  Roznowski 

Ohio  State  University 
Cynthia  Searcy 

University  of  Georgia 


External  Advisors 

Paul  Sacket 

University  of  Minnesota 
Susan  Embretson 

University  of  Kansas 
Isaac  Bejar 
ETS 
Clint  Walker 
ARI 

David  Alderton 

NPRDC  (currently  CDC) 


Cost-Benefit  Analysis 


"  Annual  Savings  to  Air  Force  from: 

Better  Selection:  $1,036,400 
Better  Job  Performance:  $1,619,000 
Total  Savings:  $2,655,400 

"  Annual  Savings  Extrapolated  to  DoD:  $14,752,200 

"  Net  Annual  Savings  Zero  when  Cost  of  Administering 
APT  reaches  $88  per  Selectee 
(Current  Cost  of  ASVAB  -  $1 29/Selectee) 


Section  3:  EQS  Model  results  using  full-sample  correlations  corrected 
for  range  restriction 


/TITLE  SAMPLE  1 

/SPECIFICATIONS 

VARIABLES=22 ;  CASES=9325 ;  METHODS=ML; 

MATRIX=CORRELAT ION ;  ANALYS I S=COVARI ANCE ; 
fields=6; 

/LABELS 

V1=GS;  V2=AR/  V3==WK;  V4=PC;  V5=N0; 

V6=CS;  V7=AS;  V8=MK;  V9=MC;  V10=EI; 

V11=WMQ4XPCA;  V12=WMS3XPCA;  V13=WMV1XPCA;  V14=SLQ3XPCA;  V15=SLS1XPCA; 

V16=SLV2XPCA;  V17=INQ3XPCA;  V18=INS2XPCA;  V19=INV1XPCA;  V20=FLQ2XPCA; 

V21=FLS1XPCA;  V22-FLV3XPCA; 

Fl=general;  F2=Gc;  F3=Gv;  F4=Tech;  F5=Quant;  F6=FactLrn;  F7=speed; 

/technical 

iterations=360; 

/print 

fit=all; 


/equations 


vl 

= 

*fl 

+ 

*f2 

+ 

*f3 

+ 

*f4 

+ 

el; 

v2 

= 

+ 

*f2 

+ 

*f3 

+ 

*f4 

+  *f5 

+ 

*f7 

+ 

e2; 

v3 

= 

+ 

*f2 

+ 

*fl 

+ 

e3; 

v4 

= 

+ 

*f2 

+ 

*f7 

+ 

e4; 

v5 

= 

+  *f5 

+ 

*f7 

+ 

e5; 

v6 

= 

*fl 

+ 

*fl 

+ 

e6 ; 

v7 

+ 

*f2 

+ 

*f3 

+ 

★  f4 

+ 

e7; 

v8 

= 

+ 

*f2 

+  *f5 

+ 

*fl 

+ 

e8 ; 

v9 

*fl 

+ 

*f2 

+ 

*f3 

*f4 

e9; 

vlO 

= 

*fl 

+ 

*f2 

+ 

*f3 

+ 

*f4 

+ 

elO; 

vll 

= 

*fl 

+ 

ell; 

vl2 

= 

*fl 

+ 

*f3 

+ 

el2; 

vl3 

= 

•f 

el3; 

vl4 

*fl 

+ 

el4; 

vl5 

+ 

*f3 

+ 

el5; 

vl6 

= 

*fl 

+ 

el6; 

vl7 

+  *f5 

+ 

el7; 

vl8 

= 

*fl 

+ 

*f3 

el8; 

vl9 

+ 

*f2 

+ 

el9; 

v20 

*fl 

+  *f5 

+ 

*f6 

+ 

e20; 

v21 

= 

*fl 

+ 

*f3 

+ 

*f6 

-  + 

e21; 

v22 

= 

*fl 

+ 

*f2 

+ 

*f6 

e22; 

/variances 

fl 

= 

1;  f2 

1; 

f3  = 

=  1; 

f4 

=  1; 

f5  = 

1; 

f6  = 

1; 

f7  =  1; 

el=*; 
e2=*; 
e3=*; 
e4=*  ; 
e5-*; 
e6=*; 
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e7=*; 

e8=*; 

e9=*; 

elO=*; 

ell=*; 

el2=*; 

el3=*; 

el4=*; 

el5=*; 

el6=*; 

el7=*; 

el8=*; 

el9=*; 

e20=*; 

e21=*; 

e22=*; 


/covariances  !  could  add  these;  but  they  are  commented  out 
!F7,F3=*; 

1F7,F4=*; 

!F6,F3=*; 

/matrix 

1.0000 


0.7217 

1.0000 

0.8008 

0.7079 

1.0000 

0.6893 

0.6719 

0.8027 

1.0000 

0.5168 

0.6282 

0.6047 

0.5970 

1.0000 

0.4516 

0.5140 

0.5501 

0.5604 

0.7025 

1.0000 

0.6370 

0.5333 

0.5291 

0.4232 

0.2961 

0.2250 

0.6947 

0.8266 

0.6698 

0.6370 

0.6236 

0.5186 

0.6948 

0.6843 

0.5935 

0.5209 

0.4037 

0.3352 

0.7601 

0.6582 

0.6839 

0.5732 

0.4144 

0.3416 

0.4403 

0.5456 

0.4633 

0.4415 

0.3857 

0.3571 

0.4837 

0.5819 

0.4363 

0.4110 

0.3561 

0.3180 

0.5070 

0.5740 

0.5529 

0.5411 

0.4547 

0.4381 

0.5063 

0.5938 

0.5023 

0.5055 

0.4170 

0.3924 

0.4720 

0.5835 

0.4445 

0.4139 

0.4159 

0.3928 

0.4645 

0.5419 

0.4917 

0.4597 

0.3952 

0.3622 

0.4604 

0.5930 

0.4625 

0.4453 

0.4388 

0.3475 

0.4856 

0.5478 

0.4445 

0.4069 

0.3441 

0.3004 

0.6584 

0.6589 

0.6929 

0.6258 

0.4923 

•0.4414 

0.3125 

0.4285 

0.3381 

0.3469 

0.3423 

0.3077 

0.4844 

0.5439 

0.4175 

0.3852 

0.3003 

0.2785 

0.4123 

0.4481 

0.4310 

0.4317 

0.3572 

0.3604 

1.0000 

0.4152 

1.0000 

0.7411 

0.6002 

1.0000 

0.7454 

0.5851 

0.7453 

1.0000 

0.3019 

0.5085 

0.4264 

0.3775 

1.0000 

0.3775 

0.5706 

0.5350 

0.4519 

0.4643 

1.0000 

0.3326 

0.5510 

0.4462 

0.4252 

0.5474 

0.5249 

0.3205 

0.5914 

0.4586 

0.4298 

0.4494 

0.5418 

0.3671 

0.5611 

0.5385 

0.4405 

0.4600 

0.5340 

0.2883 

0.5426 

0.4253 

0.3969 

0.4328 

0.4697 

0.3001 

0.5833 

0.4245 

0.3945 

0.4120 

0.4552 

0.3732 

0.5218 

0.5148 

0.4505 

0.4062 

0.5067 
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0.4303 

0.6336 

0.5275 

0.1507 

0.4248 

0.2551 

0.3899 

0.5342 

0.5544 

0.2453 

1.0000 

0.4540 

0.3279 

0.5231 

1.0000 

0.4900 

0.4805 

1.0000 

0.4844 

0.5084 

0.4513 

0.4452 

0.4530 

0.4290 

0.4318 

0.4607 

0.4566 

0.5146 

0.5179 

0.4470 

0.3622 

0.3890 

0.3520 

0.4651 

0.5034 

0.5462 

0.4791 

1.0000 

0.4532 

0.3928 

0.3707 

1.0000 

0.4563 

0.3898 

1.0000 

0.4397 

0.3993 

0.4247 

/standard 

deviations 

10.0130 

10. 

0126 

9.9964 

9. 

9885 

25.8858 

11. 

9604 

22.8380  10.8668 


0.5526 

0.4400 

0.2334 

0.3085 

0.4575 

0.4354 

0.3399 

0.3691 

1.0000 

0.4623 

1.0000 

0.4354 

0.4417 

0.4833 

0.4713 

0.3529 

0.3541 

0.4414 

0.4144 

0.4083 

0.3682 

1.0000 

9.9641 

10.0394 

10.0434 

10.0004 

21.8499 

17.5959 

25.3116 

8.4555 

0.4753 

0.3992 

0.6336 

0.4368 


1.0000 

0.4695 

0.3006 

0.4971 

0.3532 


9.9586  9.9976 
18.5389  25.5200 
17.3835  13.8977 


/wtest 

/Imtest 

/end 
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Goff  Model  fit  to  corrected  correlations  for  APT  and  ASVAB  test  batteries 
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/TITLE  SAMPLE  2 

/SPECIFICATIONS 

VARI ABLES=2  3 ;  CASES-2 270;  METHODS=ML ; 

MATRIX=CORRELAT ION ;  ANALYS I S=COVARI ANCE ; 
fielcls=6; 

/LABELS 

V1=GS;  V2=AR;  V3=WK;  V4=PC;  V5=NO; 

V6=CS;  V7=AS;  V8=MK;  V9-MC;  VIO-EI; 

V11-WMQ4XPCA;  V12=WMS3XPCA;  V13=WMV1XPCA;  V14=SLQ3XPCA;  V15=SLS1XPCA; 
V16-SLV2XPCA;  V17-INQ3XPCA;  V18=INS2XPCA;  V19=INV1XPCA;  V20=FLQ2XPCA; 
V21=FLS1XPCA;  V22=FLV3XPCA;  V23=CRS_GRD; 

/technical 

iterations=360; 

/print 

fit=all; 


/equations 


vl 

= 

*£1 

+ 

*£2 

*£3 

+ 

*£4 

+ 

el; 

v2 

= 

★  £1 

+ 

*£2 

+ 

*£3 

+ 

*£4 

+ 

*£5 

+ 

★  £7 

e2; 

v3 

= 

*£1 

+ 

*£2 

+ 

*£7 

+ 

e3; 

v4 

*£1 

+ 

*£2 

+ 

*£7 

+ 

e4; 

v5 

= 

*£1 

+ 

*£5 

+ 

*£7 

+ 

e5 ; 

v6 

= 

*£1 

+ 

*£7 

+ 

e6; 

v7 

= 

*£1 

+ 

*£2 

+ 

*£3 

+ 

*£4 

+ 

e7; 

v8 

= 

*£1 

*£2 

+ 

*£5 

+ 

*£7 

+ 

e8; 

v9 

*£1 

+ 

*£2 

+ 

*£3 

+ 

★  £4 

+ 

e9; 

vlO 

= 

*£1 

+ 

*£2 

*£3 

+ 

*£4 

+ 

elO; 

vll 

= 

*£1 

+ 

ell; 

vl2 

= 

*£1 

*£3 

el2; 

vl3 

*£1 

el3; 

vl4 

= 

*£1 

el4;. 

vl5 

= 

*£1 

+ 

*£3 

+ 

el5; 

vl6 

= 

*£1 

+ 

el6; 

vl7 

*£1 

+ 

*£5 

+ 

el7; 

vl8 

= 

*£1 

+ 

*£3 

el8; 

vl9 

*£1 

+ 

*£2 

+ 

el9; 

v20 

= 

+ 

*£5 

+ 

*£6 

+ 

e20; 

v21 

= 

*£1 

+ 

*£3 

+ 

*£6 

+ 

e21; 

v22 

*£1 

+ 

*£2 

+ 

*£6 

+ 

e22; 

v23 

= 

*£1 

+ 

*£2 

+  * 

£3 

+  * 

£4 

+  *£5 

+ 

*£6 

+  *£7 

+ 

e23 ; 

/variances 

fl  =  1;  f2  =  1;  f3  -  1;  f4  =  1;  £5  =  1;  £6  =  1;  £7  =  1; 

el-^; 

e2=*; 

e3=*; 

e4=*; 

e5=*; 

e6-*; 

e7=*; 

e8-*; 

e9=*; 

elO=*; 

ell=*; 

el2=*; 
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el3=*; 

el4-*; 

el5=*; 

el6=*; 

el7-*; 

el8=*; 

el9=*; 

e20=*; 

e21=*; 

e22=*; 

e23=*; 

/covariances 

/matrix 

1.0000 


0.7217 

1.0000 

0.8008 

0.7079 

1.0000 

0.6893 

0.6719 

0.8027 

1.0000 

0.5168 

0.6282 

0.6047 

0.5970 

1.0000 

0.4516 

0.5140 

0.5501 

0.5604 

0.7025 

1.0000 

0.6370 

0.5333 

0.5291 

0.4232 

0.2961 

0.2250 

0.6947 

0.8266 

0.6698 

0.6370 

0.6236 

0.5186 

0.6948 

0.6843 

0.5935 

0.5209 

0.4037 

0.3352 

0.7601 

0.6582 

0.6839 

0.5732 

0.4144 

0.3416 

0.3588 

0.4862 

0.3685 

0.3603 

0.3085 

0.2941 

0.4663 

0.5567 

0.4261 

0.3750 

0.3182 

0.3022 

0.5013 

0.5619 

0.5505 

0.5299 

0.4327 

0.4293 

0.4700 

0.5539 

0.4691 

0.4743 

0.3784 

0.3710 

0.4900 

0.5906 

0.4711 

0.4159 

0.4118 

0.3988 

0.4650 

0.5240 

0.4827 

0.4437 

0.3713 

0.3476 

0.4668 

0.5817 

0.4707 

0.4171 

0.4193 

0.3204 

0.4582 

0.5168 

0.4179 

0.3620 

0.2886 

0.2531 

0.6519 

0.6466 

0.6853 

0.6246 

0.4807 

0.4280 

0.2838 

0.4063 

0,3022 

0.3220 

0.3186 

0.2724 

0.4740 

0.5251 

0.4063 

0.3487 

0.2798 

0.2690 

0.4222 

0.4472 

0.4322 

0.4248 

0.3621 

0.3795 

0.6639 

0.6612 

0.6542 

0.6244 

0.5268 

0.4734 

1.0000 

0.4152 

1.0000 

0,7411 

0.6002 

1.0000 

0.7454 

0.5851 

0.7453 

1.0000 

0.2349 

0.4488 

0.3673 

0.3092 

1.0000 

0.3642 

0.5570 

0.5302 

0.4360 

0.4094 

1.0000 

0.3129 

0.5288 

0.4414 

0.4186 

0.4965 

0.4931 

0.2817 

0.5575 

0.4151 

0.3978 

0.3951 

0.4935 

0.3896 

0.5659 

0.5658 

0.4710 

0.4258 

0.5157 

0.2756 

0.5230 

0.4108 

0.3981 

0.3639 

0.4460 

0.2917 

0.5675 

0.4109 

0.3965 

0.3466 

0.4311 

0.3406 

0.5055 

0.4886 

0.4337 

0.3616 

0.5005 

0.4156 

0.6230 

0.5281 

0.5432 

0.3747 

0.4610 

0.1143 

0.4079 

0.2355 

0.2161 

0.2673 

0.3657 

0.3674 

0.5258 

0.5469 

0.4543 

0.3993 

0.6445 

0.2353 

0.4550 

0.3187 

0.3328 

0.3151 

0.4015 

0.4916 

0.6646 

0.5500 

0.6065 

0.3496 

0.4654 

1.0000 

0.4961 

1.0000 
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0.4917 

0.4662  1.0000 

0.4672 

0.4878  0.4412 

1.0000 

0.4332 

0.4084  0.4185 

0,4085  1.0000 

0.4053 

0.4179  0.4381 

0,4155  0.4357 

1.0000 

0.5207 

0.4942  0.4543 

0.4707  0.4641 

0.4513 

0.3207 

0.3532  0.3304 

0.3111  0.3143 

0.2754 

0.4591 

0.4739  0.5447 

0.4434  0.3940 
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:  deviations 

10.0130 
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9.9885  10.0434  10. 

0004  17.3387  24. 

3161 

25.4225 

11.7179  21.7519  17. 

6412  16.2118  13. 

5377 

22.5809 

10.4000  24.9964  8. 

0408  7.8028 
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/wtest 

/Imtest 

/end 
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Goff  Model  fit  to  corrected  correlations  and  extended  to  predict  final  course  grade. 
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Uncorrected  correlations  and  standard  deviations:  Sample  1  (n=9325) 
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/standard  deviations 


6.635 

8.246 

21.440 

17.098 


6.560 

6.443 

10.145 

10.062 


4.346 

8.026 

19.071 

22.727 


4.644 
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Uncorrected  correlations  and  standard  deviations:  Sample  2 
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{n=2270) 
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.242 

.271 

.261 

.268 

.171 

.223 

.317 

.344 

.293 

.264 

.220 

.264 

1.000 

.188 

1.000 

.279 

.268 

.241 

.286 

.244 

.171 

.296 

.273 

.216 

.240 

.198 
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.234 

.213 
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.240 

1.000 
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1.000 

.242 

.233 

.292  1.000 
.228  .273 
.177  .155 
.242  .356 
.187  .175 
.229  .196 


1.000 


/standard  deviations 

6.708  6.807  4.464  4.753  5.873  6.821 

8.665  6.338  8.364  8.071  15.980  21.753 

21.278  10.178  18.857  15.470  14.100  12.322 

17.150  9.745  22.753  7.187  5.843 
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